QUESTIONNAIRE DEVELOPMENT RESOURCE 


TYPE 

DESCRIPTION 

SOURCE 

Question wording 

Ambiguous or complex wording may be interpreted 
differently than intended. 

Questionnaire 

Question order 

Answers to some questions (especially those asking 
for opinions) may be affected depending on where in 
the questionnaire they are asked. 

Questionnaire 

Response-code 

precision 

Possible answers may not have an appropriate 
response code, or codes may not be mutually 
exclusive 

Questionnaire 

Length of interview 

In a lengthy interview, the respondent may tire and 
put less serious thought into his or her responses 
near the end of the interview than at the beginning. 

Questionnaire 

Recall error 

The respondent may be unable to recall factual items 
accurately. 

Respondent 

Interviewer 
clarifications to 
respondent 

Lack of thorough training and interviewing 
experience could cause the interviewer to mislead 
the respondent on questions that may not be clear. 

Interviewer 

Coding error 

The interviewer may misinterpret the respondent’s 
answer, mark an incorrect response code, or mark an 
incorrect data entry on the questionnaire. 

Interviewer 

Sloppy 

questionnaire 

administration 

The interviewer may ask a question incorrectly, 
record an incorrect response, or fail to follow skip 
instructions. 

Interviewer 

Data entry error 

In paper-administered systems, the person entering 
the data codes into computer files may inadvertently 
key in the wrong value. 

Data 

processing 
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TYPES OF MEASUREMENT ERROR 


Measurement refers to the process of obtaining the qualitative or quantitative values needed to 
meet the surveillance objectives. 

The quality of measurements can be enhanced or diminished by several factors. Responses to 
questions can be influenced by: 

• Wording, format, and order of the questions; 

• Characteristics of the respondent, such as socioeconomic background and attitude toward 
interviews; 

• Adherence to wording; and 

• Interviewer’s tone of voice, interviewing pace, and ability to maintain scientific objectivity 
when helping to clarify respondents’ answers. 

Staff members involved in editing and coding the data also contribute to the measurement 
process and, thereby, arc a source of measurement error. 

Measurement errors can be decreased if the questions are phrased properly on the questionnaire, 
read properly by the interviewer, understood and answered truthfully by the respondent, and 
checked for errors by data processors. 

Some common sources of measurement error that can affect the quality of data are listed on the 
next page. 
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Pretesting process: 

• Allow 2 to 3 days for pretesting. 

• First conduct the pretest among your interviewing staff. 

• Pretest on a sample of respondents. A large sample size provides a better sense of 
question stability. 

• Use paper questionnaires so interviewers can make notes. 

• Speak to whomever answers the telephone. 

• Tell the respondent you are conducting a test of the questionnaire. 

• Try to obtain a 50/50 ratio of males to females. For example, if respondents are primarily 
male, instruct interviewers to ask for females. 

• Do not make successive attempts to reach any given number. 

• Meet with interviewers to share notes and comments, and obtain feedback regarding their 
impressions of the instrument and any suggestions they have to improve it. 

• Ask respondents what they think about the issues you are pretesting, if possible. For 
example ask the following questions: 

- What did you think about this question? 

- What are your thoughts on the wording of this question? 

- Do you think the question on [topic X] should have been asked prior to the question 
on [topic Y]? 

- What does [a certain word] mean to you? 

- How did you interpret this question? What did this question mean to you? 


A bibliography on cognitive testing is provided in “Cognitive Testing of Questionnaires 
Bibliography” found in the resources for this section. 
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BRFSS data can be categorized to get health information on various subgroups. For example, by 
analyzing drinking data by gender, we find that men are more likely to be binge drinkers than 
women. BRFSS data can also be used to identify geographical differences in risk behaviors. 

Another strength of the BRFSS is comparability of data. Because questions are standardized, 
health care professionals can make state-by-state comparisons and track health trends over time. 


Who uses BRFSS survey results? BRFSS survey results are used by state and local health 
departments, CDC, academic researchers, health professionals, nonprofit organizations, 
insurance companies, managed care organizations, students, the media; and the military. 

State health departments use BRFSS data to create annual and periodic reports, fact sheets, press 
releases, or other publications, to educate the public, the professional health community, and 
policy makers about the prevalence of modifiable behavioral risk factors and of preventive health 
screening practices. 

Public health departments use BRFSS data to track health objectives. They rely heavily on data 
from the BRFSS to: 

• determine priority health issues and identify populations at highest risk for illness, 
disability, and death 

• develop strategic plans and target prevention programs 

• monitor the effectiveness of interventions and progress in meeting prevention goals 

• educate the public, the health community, and policy makers about disease prevention 

• support community policies that promote health and prevent disease 

Radio, television, newspapers, and state health departments use BRFSS data to inform the public 
about health risks. 

A specific example: how some of the BRFSS data were used in Oregon. The fight 
to pass workplace smoking bans has been tough in Oregon. Supporters of workplace smoking 
bans were called health fanatics Who were out of step with the wishes of ordinary citizens. The 
BRFSS produced powerful evidence that proved otherwise. The surveys revealed that about 9 in 
10 Oregonians think that breathing secondhand smoke is harmful to health and that people 
should be protected. As a result, workplace smoking bans have been passed in communities 
across the state. About 30 percent of Oregonians now live in localities where smoking is banned 
in all or nearly all workplaces. 


To find out how BRFSS data are being used in each of the participating states, go to the BRFSS 
Web site (http://www.cdc.gov/brfts/). 
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DETERMINING QUESTION VALIDITY AND 
RELIABILITY 


Validity is the extent to which survey questions actually measure what they intend to measure. 
The validity of a survey will depend on its ability to grasp the reality held by those being 
investigated, and on whether the findings are applicable (generalizable) beyond the sample 
surveyed, i.e., on the quality of the sampling design. 

Reliability is a measure of the extent to which observations of a survey are repeatable, or 
produce the same answers. Measurement unreliability may be inherent in the instrument itself 
(e.g., the wording of a question) or come from differences in procedure (e.g., the interviewer’s 
tone of voice when asking the question). A question is reliable if it produces consistent answers. 

A survey or survey item first need to be valid before it is reliable because having reliability 
without validity would not make any sense. 


Pretesting the questions will allow checking for reliability and validity. However, 
cognitive testing can be avoided by using standardized questions that have been developed, field- 
tested, and administered by others. In addition to these obvious advantages, the use of 
standardized questions permits comparisons among surveys. 

If possible, new or revised questions should be pretested at least once and then subjected to a 
pilot test prior to using them in a survey. 

During pretesting, be sure to analyze the following; 

• Ordering of questions in the survey 

Use two different surveys with different question ordering and analyze the results. 

• Question wording and clarity 

Discuss among interviewers and question authors. 

• Question appropriateness 

Analyze respondent answers and discuss with question authors. 

• Skip patterns and the adequacy of response categories 

Compare respondent answers and discuss among interviewers and authors. 
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HOW BEHAVIOURAL RISK FACTOR SURVEILLANCE 
SYSTEM DATA ARE USED 


What is the BRFSS? In 1984, the Centers for Disease Control and Prevention (CDC) 
initiated the state-based Behavioral Risk Factor Surveillance System (BRFSS) to collect 
prevalence data on risk behaviors and preventive health practices that affect health status. States 
conduct monthly telephone surveillance using a standardized questionnaire to determine the 
distribution of risk behaviors and health practices among adults. Responses are forwarded to 
CDC, where the monthly data are aggregated for each state, returned with standard tabulations, 
and published at year's end by each state. 

People aged 18 and older are randomly called and asked to take part in the survey. They remain 
completely anonymous. 

Questions focus on health behaviors related to several leading causes of death and disease, for 
example: 

• using condoms to prevent the spread of AIDS 

• taking medication for high blood pressure 

• smoking or using tobacco 

• getting a mammogram 

• not exercising on a regular basis 


How are BRFSS data used? As an effort to promote healthy personal behaviors, thereby 
preventing disability and premature death, the BRFSS provides data that arc used to: 

• identify and address emerging health issues 

• identify demographic differences 

• document health trends 

• measure trends in health-related behaviors 

• compare health behaviors across states 

• publish scientific articles in professional journals 

• educate the public 

• benefit health research and improve public health strategies 

• assess risk for chronic diseases 

• design and monitor health intervention and services 

• formulate policy and propose legislation for health initiatives 

• measure progress toward achieving state and national health objectives 

Data gathered through the BRFSS provide important information for the development of public 
health programs. 
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